[transformers snippet] Support pipeline VLMs - #1012

The example is super nice, but not all image-text-to-text models support multiple images reliably. I'd go for a simpler single-image example for now.

Agreed with this! another problem is that although most models use the <image> token, some models use a different token. For example, Pixtral uses [IMG], and mllama uses <|image|>, so this snippet would not work with these models.

However since you have this check:

if (model.tags.includes("conversational") && model.config?.tokenizer_config?.chat_template) {

It would be even better and simpler to use chat template for vlms also, with something like this:

image_ny = "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
image_chicago = "https://cdn.britannica.com/59/94459-050-DBA42467/Skyline-Chicago.jpg"
messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "What’s the difference between these two images?"},
            {"type": "image"},
            {"type": "image"},
        ],
    }
]
outputs = pipe([image_ny, image_chicago], text=messages)

I completely agree with @yonigozlan best to have chat template

I think this was finally handled by @Vaibhavs10 in #1434 (sorry, I did not remember about this PR when the other one was opened)